Papers with multiple-choice tasks
Probe-Free Low-Rank Activation Intervention (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing activation intervention methods involve an activation probe to detect undesirable generation, triggering the activation modification to steer subsequent generation. |
| Approach: | They propose a probe-free intervention method FLORAIN for all attention heads in a specific activation layer. |
| Outcome: | The proposed method outperforms baseline methods in enhancing model truthfulness and quality across generation and multiple-choice tasks. |
ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding (2025.findings-naacl)
Copied to clipboard
Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Yonas Chanie, Bontu Fufa Balcha, Negasi Haile Abadi, Henok Biadglign Ademtew, Mulubrhan Abebe Nerea, Debela Desalegn Yadeta, Derartu Dagne Geremew, Assefa Atsbiha Tesfu, Philipp Slusallek, Thamar Solorio, Dietrich Klakow
| Challenge: | Large language models (LLMs) evaluation is gaining increasing attention as they are typically trained on general-domain datasets while demonstrating notable performance on tasks out of their training domains. |
| Approach: | They propose an LLM evaluation benchmark for low-resource languages that focuses on low-rsource language understanding in culture-specific scenarios. |
| Outcome: | The proposed benchmarks outperform monolingual evaluations on proverb generation tasks and native language proverb descriptions on multiple choice tasks. |
Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective (2022.emnlp-main)
Copied to clipboard
Ping Yang, Junjie Wang, Ruyi Gan, Xinyu Zhu, Lin Zhang, Ziwei Wu, Xinyu Gao, Jiaxing Zhang, Tetsuya Sakai
| Challenge: | Existing approaches to zero-shot learning are format-agnostic and can address new learning tasks without additional training. |
| Approach: | They propose a new paradigm for zero-shot learning that is format agnostic and compatible with any format and applicable to a list of language tasks. |
| Outcome: | The proposed model shows state-of-the-art performance on several benchmarks and produces satisfactory results on tasks such as text classification and commonsense reasoning. |
Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are used to discriminate on multiple-choice tasks that place probability mass on vocabulary tokens that aren’t among the given answer choices. |
| Approach: | They propose a mathematical formalism for SFC which allows us to quantify and bound its impact for the first time. |
| Outcome: | The proposed method eliminates the impact of SFC in the majority of instances. |
FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue (2022.emnlp-main)
Copied to clipboard
Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lise Getoor, Jay Pujara, William Yang Wang
| Challenge: | Prior studies of task transfer in dialogue consider only 2-4 tasks, focus on multitasks. |
| Approach: | They propose a benchmark for FEw-sample TAsk transfer in open-domain dialogue. |
| Outcome: | The proposed benchmark analyzes the transferability between 132 source-target task pairs and provides a baseline for future work. |
Training Trajectories of Language Models Across Scales (2023.acl-long)
Copied to clipboard
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. |
| Approach: | They analyze the training checkpoints of different-sized OPT models on next-token prediction, sequence-level generation and downstream tasks. |
| Outcome: | The results show that language models of different sizes learn more during training . small models halt at hallucinations, larger ones learn to assign lower probabilities . |
VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent benchmarks release only training and validation sets, keeping the test set labels closed-source. |
| Approach: | They propose to extract variables from each test case and define a value range for each variable. |
| Outcome: | The proposed method improves the accuracy of the evaluations on four datasets covering mathematical generation and multiple-choice tasks. |
Unveiling Cultural Blind Spots: Analyzing the Limitations of mLLMs in Procedural Text Comprehension (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks, including text summarization, multi-modal machine translation, and code generation and understanding. |
| Approach: | They propose a benchmark to evaluate mLLMs’ ability to process and reason over culturally diverse procedural texts in multiple languages. |
| Outcome: | The proposed benchmarks show that mLLMs struggle with culturally contextualized procedural content, especially in low-resource languages, and perform better on multiple-choice tasks presented in conversational formats than on direct questions. |
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2025.findings-acl)
Copied to clipboard
Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen
| Challenge: | Recent studies have identified significant redundancy in large language models . quantization and pruning are two methods that reduce computational resources . |
| Approach: | They propose simple pruning methods that prune redundant layers based on their BI scores. |
| Outcome: | The proposed pruning methods demonstrate superior performance over previous pruning methods. |
Beyond Generation: Leveraging LLM Creativity to Overcome Label Bias in Classification (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to mitigate label bias by leveraging in-domain data are often unavailable in real-world scenarios. |
| Approach: | They propose a calibration method that generates synthetic in-domain data from a few in-context demonstrations and utilizes it for calibration. |
| Outcome: | The proposed method reduces label bias by leveraging in-domain data from demonstrations. |
Evaluating Cultural Knowledge and Reasoning in LLMs Through Persian Allusions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Allusion recognition is a critical test of LLMs' ability to deploy stored information in open-ended, figurative settings. |
| Approach: | They propose a framework for evaluating Persian literary allusions through annotations and LLM-generated texts incorporating allusion in novel contexts. |
| Outcome: | The proposed framework evaluates Persian literary allusions through annotations and LLM-generated texts incorporating allusion in novel contexts. |